Neural Compatibility Modeling with Attentive Knowledge Distillation


  Recently, the booming fashion industry and the huge potential benefit has attracted tremendous attention from many research communities. In particular, the problem of complementary clothing matching has gained increasing research efforts as matching clothes to make a suitable outfit has become a daily headache of many people, especially those who do not have a good taste of aesthetics. Thanks to the remarkable success of neural networks in various applications such as image classification and speech recognition, most of existing works adopt this pure data-driven learning method to analyze fashion items. Apparently, existing works overlook the rich valuable knowledge (rules) accumulated in fashion domain, especially the rules regarding clothing matching. To bridge this research gap, in this work, we aim to tackle the problem of complementary clothing matching by integrating the advanced deep neural networks and the rich fashion domain knowledge. In particular, considering that the rules can be fuzzy and different rules may have different rule confidence to different samples, we propose a neural compatibility modeling scheme with attentive knowledge distillation based on a teacher-student network scheme. Extensive experiments on real world dataset show the superiority of our model over the state-of-the-art methods, based on which we also provide certain fashion insights that can benefit the future research.


Figure 3: Illustration of the proposed scheme. The student network, consisting of dual-path neural networks, aims to learn the latent compatibility space where the implicit preference among items can be modeled via Bayesian Personalized Ranking. The teacher network encodes the domain knowledge and guide the student network via attentive knowledge distillation. t: top, b: bottom. “->”: category hierarchy. “>”: pair-wise preference. “no”: negative rules. The width of the arrows originated from rules refers to the rule confidence.


• We propose an attentive knowledge distillation scheme, which is able to encapsulate the fashion domain knowledge to the traditional neural networks and hence boost the model performance. To the best of our knowledge, this is the first attempt to incorporate fashion domain knowledge to boost the compatibility modeling performance in the context of clothing matching.

• Considering that different knowledge rules can have different confidence in the knowledge distillation procedure, we introduce the attention mechanism to the proposed scheme to flexibly assign the rule confidence.

• Extensive experiments are conducted on the real world dataset and the results demonstrate the superiority of the proposed scheme over the state-of-the-art methods. Apart from numerical results, we also provided several deep insights based on the experimental results.


  In this work, we use the publicly released dataset FashionVC to evaluated our proposed model. FashionVC consists of 20,726 outfts with 14,871 tops and 13,663 bottoms, collected from the online fashion community Polyvore, where a great amount of outft compositions shared by fashion experts are publicly available. Each fashion item (i.e., top or bottom) in FashionVC is associated with a visual image, categories and title description.



• extract_rule.py: To determine whether the sample satisfies the rules and return a result.

• matching_class.py: To implement MLP and CNNs in classes.

• matching_attention.py: To train the model and calculate performance.

Environment requirements

  The code is written in Python (2.7) and Theano(0.9).


  Run 'matching_attention.py'.

Download code